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ABSTRACT 



A computer program product for detecting eye color defects 
of a subject in an image due to flash illumination comprises: 
a computer readable storage medium having a computer 
program stored thereon for performing the steps of detecting 
skin colored regions in a digital image; searching the skin 
colored regions for groups of pixels with color characteristic 
of redeye defect; and correcting color of the pixels based on 
a location of redeye defect found in step (b). 
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COMPUTER PROGRAM PRODUCT FOR FIG. 3 is a detailed flowchart of the continuous skin 

REDEYE DETECTION colored region determination portion of FIG. 2; 

FIG. 4 is a binary representation of FIG. 1 illustrating 

HELD OF THE INVENTION skin-colored regions; 

The invention relates generally to the field of digital ^ FIG. 5 is a detailed viewed of the individual continues 

image processing and, more particular to a method for colored regions of FIG, 4; 

detecting redeye in digital images. FIG. 6 is a diagram of ellipses fitted to the views of FIG. 

5; 

BACKGROUND OF THE INVENTION piG. 7 illustrates resized candidate face regions; 

When flash illumination is used for the capture of an FIG. 8 is a diagram of resized ellipses corresponding to 

image sometimes the pupils of people in the image appear the candidate face regions fitted to FIG. 7; 

red. This is caused by light from the flash unit entering the FIG. 9 is a detailed flowchart of the candidate redeye 

pupil, muhiply reflecting off the retina, and finally exiting determination portion of FIG. 2; 

back througji the pupil. Because light is partially absorbed is FIG. 10 illustrates the candidate redeye defects of FIG. 7; 

by capiUaries in the retina the pupil appears red in the image. FIG. U is a detailed flowchart of the eye detection portion 

This phenomena is referred to as "redeye." The probability jtjq 2* 

of redeye being observed increases the closer the flash unit r?ir^ n n ♦ * * i * j j 

i . , . ^ L , ^ jx • FIG. 12 illustrates an eye template, and zone map; and 

is to the optical axis of the lens. Therefore, redeye is ^„ -n * . - cl r.u : • 

commonly observed in images captured by a smaU camera 20 ^ ^^"""^ °^ P'"'"" 

with an integral flash unit, 

Commonly assigned U.S. Pat. No. 5,432,863 describes a DETAILED DESCRIPTION OF THE 

user-interactive method for the detection of objects in an INVENTION 

image that have the color characteristic of redeye. This In the following description, the present invention will be 

method automatically detects candidate redeye pixels based 25 described in the preferred embodiment as a software pro- 

on shape coloration and brightness. gram. Those skilled in the art will readily recognize that the 

Although the presently known method of detecting redeye equivalent of such software may also be constructed in 

is satisfactory, it is not without drawbacks. The method of hardware. 

U.S. Pat. No. 5,432,863 does not determine whether the FIG. lis a grayscale image 10 of a color image illustrating 

candidate pixels are located in a face or are part of a human ■^^ two pairs of redeyes 20. 

eye. Referring to FIG. 2, there is illustrated an overview 

Consequently, a need exists for detecting redeye that flowchart of the present invention. A color digital image is 

overcomes the above-described drawbacks. input to the software program residing on a computer 

system, such computer systems being well known in the art. 

SUMMARY OF THE INVENTION The code vahies of the digital image are preferably propor- 

™_ ^ . ■ J- . J . • tional to the log of the amount of exposure of the film used 

The present invenUon is directed to overcomme one or ^^^^^^ ui^ lxj^ kjl ui uiuuui ^^^j^^^i ili 

more of the problems set forth above. Briefly suomiarized, f ^^a^' ^f ' ^'^f^ 

according to one aspect of the present invention, the inven- ^^^^^^ '^^2^784 ^ ^^^^^^^^ 

tion resides in a computer program product for detecting eye 40 ^6^^°^^ e image . , , „ , 

color defects of a subject in an image due to flash Referrmg to FIG. 3, there is illustrated a detail flowchart 

illumination, comprising: a computer readable storage of step S4 in FIG. 2. First, the red, green and blue values of 

medium having a computer program stored thereon for ^he color unage are converted mto LST color space S4a 

performing the steps of: (a) detecting skin colored regions in ^^^"S ^ac relations: 
a digital image; (b) searching the skin colored regions for 45 

groups of pixels with color characteristic of redeye defect; ^ _ _L(/? + c + 5) 

and (c) correcting color of the pixels based on a location of VS" 

redeye defect found in step (b). s- ^ R b 

It is an object of the present invention to provide a method yfi 

for automatically detecting redeye defects. 50 ^ 

It is an object of the present invention to provide a method ^ ~ ^ 
for determining whether candidate redeye defects are part of 
the human face. 

It is also an object of the present invention to provide a ^^^.'''^ 8^^°.' "'"^ ""^"^ 

method for determining whether candidate redeye defects oflf^el in the color unage^ respectively. 

. r I °^ The next step is to build a three-dimensional histogram. In 

are part oi the human eye. , ... n t t ■ .1 t o 1 

, , _ ^ order to reduce the size of the histogram, first, the L, S, and 

These and other aspects, objects, features and advantage^ ^ code values are quantized by dividing them by 8.0xsqrt(3), 

of the present mvenUon wiU be more clearly understood and 2.O, and 2.0, respectively S4b. These quantized code values 

appreciated from a review of the foUowing detailed descrip- ^^^^^^ ^ combination of V, S\ 

tion of the preferred embodiments and appended claims, and p ^^^^^ ^ ^^^^^^^ ^^^^ «t,in" of the histogram. The 

by reference to the accompanying drawings. ^^^^ histogram H(L', S\ T) S4c is equal to the 

BRIEF DESCRIPTION OF THE DRAWINGS ^^^'^^ P^'^^^ ^ ^^""^ quantized code 

values of L', S', and T. An alternative way of stating this is 

FIG. 1 is a diagram illustrating redeye; 55 that the histogram tell us the number of pixels in the image 

FIG. 2 is an overview flowchart of the software program that fall into eadi bin. This number is referred to as the value 

of the present invention; of the bin. 
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The histogram is smoothed S4^ by replacing the value of Referring to FIG. 2, and as illuslrated in FIG. 5, a sub-map 

each bin by a weighted average of the value of that bin and of each skin colored region is formed by cutting out from the 

the values of immediate neighboring bins. Next, the peak skin map (FIG, 4) the smallest rectangular section that 

values in the histogram are found S4e and each bin in the contains all of that skin region S6. For example, skin region 
histogram is assigned S4/ the peak value that is located 5 306 in FIG. 5 corresponds to skin region 30fl in FIG. 4. FIG. 

closest to it. Finally, since each pixel in the color image has 5 shows the map of each separate continuous skin colored 

been assigned to a bin of the histogram and each bin has regions as an individual sub-map. The column and row of 

been assigned to a peak, a peak is assigned to each pixel in the skin map that correspond to the top left corner of the 

the color image S4g. The single band image in which a sub-map are referred to as Col^^„„, and Row^„,^p respec- 

pixel*s code value is equal to the number of the peak that it lively. In the sub-map code values of 255 (white) indicates 

was assigned to is referred to as the segmented image. that the pixel is located at a position at which skin color is 

Continuous regions in the segmented image that have the present. A code value of 0 (black) indicates the absence of 

same code value are likely to correspond to an object or part skin color. 

of an object in the color image. A unique number (label) is Referring to FIG. 2, and as illustrated in FIG. 6, in the next 

assigned to all such regions in the segmented image S4/t. step an ellipse 35 is fitted S8 to the individual skin color 

The numbers are sequentially assigned starting with 1 for the sub-maps found in step S6 (FIG. 5). A method of fitting an 

region with the greatest number of pixels. The single band ellipse to a binary image is described in Computer and Robot 

image in which code values correspond to the label of the Vision, Volume /, by Robert M. Haralick and Linda G. 

region that the pixel belongs to is called the labeled image. Shapiro, Addison-Wesley (1992), pp. 639-658. A human 

The program then decides which of the continuous face is approximately eUiptical. Therefore, if the skin color 
regions in the segmented image corresponds to a region in ^0 sub-map is of a human face, then the elUpse should fit the 

the color image that has a color that is typical of human skin. ^f^^ °*fP '^^^\^^ the minor axis of the eUipse should 

-nie average L, S, and T code values of each region is appioxunately equal the width of the face. A me^^^ 

calculated and, based on this, each region is assigned a score *^^P^ ^^*^-°^^P 5^^^° 
^skin S4i. A high value of P,jt« indicates that the region is of 

a color that is typical of human skin. Alternatively, a low ^ Ht = ifz- - ^ " ^'^ ] 
number indicates that the color of the region is atypical of 
skin. Regions for which exceeds a threshold T^j^ of 

0. 10 are referred to as skin-colored regions S4/ where N is the number of skin colored pixels (code value 
One final step is necessary to associate each face in the 255) in the map, N^, is the number of skin colored pixels 

color image with a single sldn colored regioa The process that fall outside the ellipse, N(„ is the number of skin colored 

described above will often result in a single face being pixels that are inside the ellipse, and A is the niunber of 

associated with more that one skin colored region because pixels in the ellipse. A is also referred to as the area of the 

due to complexion, shadows, and etc., the color of the face ellipse. If all of the skin colored pixels are in the ellipse and 

is not uniform. Two skin colored regions are merged into a number of skin colored pixels equals the area of the 
single skin colored region if two conditions are satisfied S4t. ^5 ellipse then Fit is equal to one and the fit is perfect. When 

The first condition requires that the two regions be inter- colored pixels fall outside of the eUipse or the area of 

connected. A pixel in region i has a connection to region j if ellipse is greater than the number of skin colored pixels 

a pixel belonging to region j is one of the eight nearest "^^ide it then the value of Fit is diminished. If the value of 

neighbor pixels. A function Q(i, j) is calculated which is ^ l^^s than a predetermmed value MmEllipseFit which 

proportional to the number of connections between pixels of is set equal to 0.70 then we conclude that the skin colored 

region i and j. The function is normaUzed so that Q(i, i) is ^ ^ process it further SIO. 

equal to 1.0. If Q(i, j) exceeds the threshold MinMerger- Another indication of whether the skin color sub-map is 

Fraction regions i and j wUl be merged into a single region ^.^^^^ ^ ^^^^^ ^^^P^ AspeclRatio which 

if the second condition is also satisfied, for example a ^ given by 
threshold of 0.005 may be used. The second condition is that 

the distance between the colors of the regions i and j given AspectRatio - ^"^^ 

by "^'^ 

must be less than MaxMergeColorDistance which is set 50 where D_,,, is the major axis of the empse and D , ^ 

e ual to 40 0 mmor axis m pixels. If AspectRaUo is greater than MaxAs- 

^™ * . , . 1 J - . . pectRatio which is set equal to 3.0 the sldn colored region 

The process of mergmg skin colored regions begins with ^ . , i_- . - .t. • . - . 1 j 

11 ^ • I.- t_ -r.u *- j-f- *- J corresponds to an object m the image that is too long and 

lhesmallestregionwmch,if the two conditions are satisfied, . , c rJi ,t . . 1 • 

A -^Z \ ^ ' ic ^ • ■ A •♦u thm to be a face. The program determines that the skin 

is merged with a larger region. If region i is merged with , , . . . r j j . n 

, ^ . . *u u *u T • • * J colored region is not a face and does not process it further 

larger region j it may then happen that region j gets merged 55 ^ *^ 

with an even larger region k. When this occurs regions i, j, ,^7".. , - , . . t j ^ 

J 1 J • * • 1 xr * * • ' A If the skin sub-map has an acceptable degree of fit to an 

and k are merged mto a single region. Note that regions land ■ „. „. \ ° . 

1 L J * *L L T ^ ellipse and the empse has an acceptable aspect ratio, the map 

k may be merced together even though the above two /\. ^ - : e ^i * 1 

* 4- c J r *u *, - potenUally mdicates the position of a face. Next, we calcu- 

conditions are not satisfied for these two regions. They are - c . o u- 1. ■ • u *u r n 

Au f *u • * 1 *• T • • Jate a resize factor S„.^/, which is given by the following 

merged because of their mutual connection to region j. 60 . pnscaie & j & 

The result of skin color detection is a map of the skin ^ 

colored regions in the color image S41. Areas that are not 

skin colored are given a code value of zero. The separate = ^'•^">'^'^'><''<^'^'^hEyeDisra«ceRa,io 
continuous skin colored regions are numbered consecutively 
in order of decreasing region size beginning with the number 65 

1. FIG. 4 shows a map of the skin colored regions in FIG. where Aim EyeDistance which is set equal to 75 pixels is the 
1. desired distance between eyes, and FaceWidthEyeDis- 
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tanceRatio which is set equal to 2.0 is the ratio between the 
width and eye distance for a typical face. If S^,^^^^ is less 
than MinPrescale 010 or greater than MaxPrescale 1.50 the 
skin colored region is not processed further SIO. The next 
step is to cut-out from the color image a sub-color-image 
that corresponds exactly to the location of the sub-map S12. 
If the minor axis of the ellipse is approximately equal to the 
width of the face then the distance between the eyes in the 
face should be close to AimEyeDistance. FIG. 7 shows the 
sub-color-images 40 after they have been resized in this 
manner. It is instructive to note that FIG. 7 is illustrated as 
a gray scale drawing, although the actual image is a color 
image. FIG. 8 shows the ellipses 50 that correspond to each 
of these sub-color-images that have also been resized S14. In 
practice, it is desirable to add extra rows and colimias to the 
edges of the resized sub-color-images and sub-maps so that 
when these images are processed further an out-of-bounds 
pixel is not addressed. Tlie top and bottom of the images are 
padded with Pad rows and the left and right side with Pad 
columns. 

Now that skin colored regions that have the shape of a 
face have been identified, the location of candidate redeyes 
need to be identified SI 6, which is illustrated in detail in 
FIG. 9. Now referring to FIG. 9, the sub-color-images 40 are 
processed so as to identify small red features. The program 
begins by defining a new single band image SI 6a with pixel 
values X given by 

^=R-Max(G^) 

where R, G, and B, are the red, green, and blue code value 
of the sub-color-image, respectively. 

Redeyes in the new image will appear as small elliptical 
areas of high code value possibly with a small low code 
value region in the middle that is due to glint in the pupil. 
The affect of glint is removed by performing a gray scale 
morphological closing S16b using a W_close x W_close 
kernel, for example a 3x3 kcrnal although other sizes may 
also be used. Gray scale morphological operations arc 
disclosed in Image Analysis and Mathematical Morphology 
Volume 1, by Jean Serra, Academic Press (1982), pp. 
424-478. Next, the small regions of high code value are 
removed by a gray scale morphological opening operation 
using a W_openxW_open kernel, for example a 5x5 kemal 
although other sizes may also be used S16c. The opened 
image is then subtracted from the closed image in order to 
form a residual image S16^. This image shows what was in 
the opened image, but not in the closed image. Namely, 
small regions of high code value which correspond to small 
red features in the sub-color-image. Next, the residual image 
is smoothed S16e with a linear filter having the kernel shown 
below. 

121 
242 
121 

For each pixel in the smoothed residual image, a 7x7 
window centered at that pixel is examined. If the code value 
of that pixel exceeds the threshold Tpeak which is set equal 
to 5 and is greater than or equal to the code value of all the 
other pixels in the window, that pixel is classified as a peak 
S16/. FIG. 10 shows the peaks 37 for all of the sub-color- 
images in FIG, 7. After all the peaks in the smoothed 
residual image have been foimd the individual peaks are 
examined S16g. First, if a pixel has been classified as a peak 
and a neighboring pixel that is west, north-west, north, or 
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north-east of this pixel has also been classified as a peak, the 
peak is eliminated SI 6k 

A pixel that has been classified as a peak is a candidate 
redeye pixel. It is possible however that the location of the 
peak coincides with glint in the pupil and not the red defect. 
For this reason, pixels within a distance GUntRadius equal 
to 2 from the peak are examined SI 6i. The candidate redeye 
pixel is moved to the nearby pixel with the highest color 
score Pco/or which will be defined below. 

Next, the candidate redeye pixel is used as a seed to grow 
a continuous region of pixels of simular color. If the number 
of pixels in the region is less than MinSize or greater than 
MaxSizc the region is not of a size that is characteristic of 
a redeye defect and the candidate redeye pixel is eliminated 
S16j. 

The result of the above processing is a map of candidate 
redeye pixels for each sub-color-image S16k. The ellipses in 
FIG. 8 are approximate maps of the region in the corre- 
sponding sub-color-images in FIG. 7 that have been identi- 
fied as potentially being a face. Therefore, only the candidate 
redeye pixels that fall inside of the ellipse are considered in 
the next phase eye detection which is outlined in FIG. 11. 

Referring back to FIG. 2, the purpose of eye detection is 
to determine whether the candidate redeye pixels are indeed 
part of an eye. The eye detection procedure requires a 
moiKDtone version of the color image 818. The green band of 
the color image is used after the contrast is increased by 
transforming the green pixel code values using the equation 
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where G is the code value of the green band and y is a 
parameter which is set equal to 2.0. This monocolor version 
of the color image will be referred to as the limiinance 
image. 

The eye detection procedure 820 in FIG. 2 is based on the 
process of template matching. It facilitates understanding to 
note that any image of an eye can be used a the template. The 
top image 60 in FIG. 12 shows a left-eye template. The 
bottom image 70 shows a division of the template into 
zones. Zone 1 is the eyebrow region. Zones 2 and 3 are the 
left and right sides of the eye, respectively. Zone 4 includes 
the pupil and iris. Zone 0 is not used. The eye template was 
taken from an image in which the distance between the eyes 
is TemplateEyeDistance equal to 306 pixels and the tUt of 
the two eyes is close to zero. As discussed above, a pair of 
redeyes in the resized color sub-images should be approxi- 
mately a distance AimEyeDistance (75 pixels) apart. 
Therefore, in order for the template to be of the proper size 
to match an eye is must be resized by a factor of 



5b = 



55 



AimEyeDistance 
TemplaieEyeDisrance 



In practice, the estimation of the face width from the 
minor axis of the ellipse will not always be accurate. Also, 
the eyes may be tilted. For this reason starting with the 

60 original left-eye template and the zone map, a collection of 
left-eye, right -eye (mirror image of left-eye), and zone maps 
are generated that span a range of sizes and orientations 822. 
The original eye template and zone map are resized from a 
factor of SoxNarrow to SoxWide in increments of SStep. 

65 Preferred values of Narrow, Wide, and Sstep are 1.5, 0,50, 
and 0.05, respectively. In order to accommodate tilt for each 
resize factor, a series of tilted templates and zone maps are 
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generated that range fironi -MaxTilt degrees (clock- wise tilt) 
to MaxTilt degrees in increments of TStep degrees S22. The 
preferred value of MaxUlt is 30 degrees and of TStep is 2.0 
degrees. 

Referring to FIG. 11, a detailed flowchart of step S20 of 5 
FIG. 2 is shown. A pair of candidate redeye pixels are 
considered that hypothetically belong to a left and right 
redeye pair S20a. The scale of the eye relative to the original 
eye template is related to the distance S20b between the 
candidate redeye pixel pair by the equation 

'""^ TemplateEyeDistance 



8 

where p e Z means that column p is in zone Z, 1 e Z means 
that row 1 is in zone Z, and is the number of pixels in the 
zone. The mean code value of the template in zone Z given 
by 

* pci /at 



where Lp (R^,) is the column of the left (right) candidate 
redeye pixel, L| (R^ is the row of the left (right) candidate 
redeye pixel. (The column numbers begin with 1 and 
increase from left to right. The row numbers begin with 1 
and increase from top to bottom.) The tilt S20i? between the 
candidate redeye pixels is given by 



is also calculated. In addition, the standard deviation of the 
template in zone Z is calculated according to the equation. 

\ ^pez/ez J 



As discussed above, an ensemble of eye templates and 
zone map templates were made that span a range of resize 
factors from SQxNarrow to S(jx\V^dc with resolution SStep 
and with a tilt from -MaxTilt degrees to MaxTilt degrees 
with a resolution TStep. The left-eye template, right-eye 
template, and zone map that most closely match the value of 
Spair and Tilt for the pair of candidate redeye pixels is used 
in tiie correlation step that follows. If or Tilt are outside 
of this range, this pair is not processed further S20c. 

After an eye template has been selected the next step is to 
determine if the region around the redeye pixel matches an 
eye. This is done by performing a correlation of the left-eye 
template with a region around the left candidate redeye pixel 
and the right-eye template with a region around the right 
candidate redeye pixel of the luminance image S20d. One 
step of the correlation process is to match up pixels of the 
template and luminance image and calculate the product of 
their code values. The center of the template images corre- 
sponds to the center of the eye. Since the candidate redeye 
pixels are close, but not necessarily at the center of an eye, 
we perform the correlation several times with the center of 
the template matched to all of the pixels within a square that 
extends a distance LookAround eqiial to 3 about the candi- 
date redeye pixel. The correlation is performed separately 
for zones 1 through 4 of the template (see FIG, 12). These 
correlations are referred to as Czl, Cz2, Cz3, and Cz4. Id 
addition, an overall correlation is calculated for a region that 
consists of the sum of zones 1 through 4. This overall 
correlation is referred to as C. The pixel in the square around 
the candidate redeye pixel with the highest value of the 
overall correlation C is the best guess of the center of an eye 
which contains the candidate redeye pixel. This pixel is 
referred to as the eye-center pixel. Both the left and right 
candidate redeye pixels have an associated eye-center pixel. 

The correlation process is now explained in detail. The 
template image is denoted by the function O (pji) where p is 
the column number and I is the row number. The number of 
columns and rows in the template is w and h, respectively. 
The center of the eye template is approximately the location 
of the center of the eye. A zone of the template is correlated 
with the luminance image which we denote by r(p,l) at 
column p^ and row l^ by calculating the product n given by. 



Similarly, we calculate the mean code value of the lumi- 
nance image in zone Z using the equation 

and the standard deviation using the follovidng equation 

Using the quantities defined above the correlation of the 
luminance image with the template in zone Z is given by the 
relation 

n - M^^f^ 

If the code values of the image and the template are exactly 
the same in zone Z then is equal to 1 .0. If the image and 
the template are completely uncorrelated then will be 
equal to zero. 

The values of C, C^j, C22, C^3, and C^^ for the eye-center 
pixels are used in the calculation of a score that is a measiu^ 
of the likelihood that the pair of candidate redeye pixels are 
part of a redeye defect in the sub-color-image S20e. Each of 
the correlations are used as a variable in an associated 
scoring function that ranges from 0.0 to 1.0. For example, 
the scoring function associated with the overall correlation 
C which we refer to as pO(C) is 0.0 if the value of C for an 
eye -center pixel indicates that it is very unlikely that the 
pixel actually is located at the center of an eye. On the other 
hand, if the value of C is in a range that is typical of the 
correlation of the template with an eye then pC(C) is 1.0. 
Otherwise pC(C) takes on an intermediate value. Hie scor- 
ing function pC(C) and o±er scoring functions described 
below are shown in FIG. 13. 

Scores are defined based on these scoring functions which 
will be combined later into an overall score for a candidate 
redeye pair. The following equation defines a score P^^^^ 
related to the overall correlation C as simply 
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The score P^^^ associated with the zone correlations is a with a redeye defect in the color image be incorrectly 
weighted average of the zone correlation scoring functions. classified. One method of confirming thai a pair of redeyes 
It has been found that the correlation in zone 4 (the pupil) is has indeed been located is to use the fact that a human face 
a much more reliable indicator of the presence of an eye than is approximately symmetric about a line that bisects the face 
the other zones. For this reason it ts given more weight than 5 S24 in FIG. 2. In order to do this, the sub-color-image is 
other zones. Typical we set the weight W equal to 6.0. rotated so that the tilt of a line connecting the best pair of 

is given by candidate redeye pixels is equal to zero. Next, an image 

centered at the midpoint between the eyes is cut-out of the 
pC^jjCzi) ^ pC^jC^) 4- pc^{Co) + y^pc^jC^) sub-color-image. This image has a width of 1.5 times the 

M/ ^- 3 10 distance between the candidate redeye pixels and a height 

equal to a quarter of its width. This image is in turn cut in 
It has been found that the standard deviation of the ^^[' ^^J":^ half-image we refer to as E^efi(p,l) and the 
himinance image that was calculated in the process of "6^* half-unage by E-nght(p4) where the superecnpt x 
calculating the overall correlation C is a good indicator if the ^^^^^ ^ ^^^'^"^ ?^ color miage. For example, EleftCpJ) 
feature in the luminance image centered at the eye-center n^fers to the red band of the imagp. The columns m 
pixel is actually an eye. For instance, if is very low than ^^^-^^%<^ are mvcrted (the first column becomes the last 
the feature is of too bw contrast to be an eye. With this in '^^^'''''^^^^^^ '^f^ 

mind we define a score associated with by correlation of E^eft(p4) and E;nght(p,l) is performed by 

first calculating the sum of products 

Finally, the color of the candidate redeye pixel must be "^>« = AfZZ^*^^^' ^^"shKp. t) 

indicative of a real redeye defect. For this calculation the ' 
red, green, and blue, code values of the candidate redeye 

pixel is convert into luminance (Lum), hue (Hue), and 25 where the summations over p and I are over all of the 

saturation (Sat) values. Luminance is calculated as follows columns and rows in die half-images, respectively, and N is 

the number of pixels in the half-images. The correlation is 

_ Max(Af, c, 5) + Miii(/?, c, B) given by 



The value of Lum for a pixel ranges from zero to the highest 
possible code value. The saturation given by 

Max(W G B)-y^R G B) where M^eft and M'^right are the mean code values of band 

Sal = 100 \{an{R G B) — ^ — ^ half-images and o^eft and o'right are the standard 

deviations. A score Pjy„ is defined based on a symmetry 
scoring function pSym(C*yy^ by 

is a value ranging firom 0 to 100. The hue is defined as in 

Computer Graphics Principles and Practice 2nd ed., P^=pSyiiiCcrsym)pSym(C^ym)pSymCd'syin) 
Addison-Wesiey Publishing Company, page 592, except the 40 gu^l score P is simply the product of P^^ and P^>. 

color red is shifted to a hue angle of 120 degrees. The value 
of Hue may range from 0 to 360 degrees. The score that 

related to the color of the candidate redeye pixel is defined ,r.i.. u-u— u*. nnjin a 

^ *^ If this score which may range between 0.0 and 1.0 exceeds 

^ a threshold MinScore which is set equal to 0.05 S26, then the 

P«rfor=pL(Luni)pH(Hue)pS{Sat) '^^ candidate redeye pixel pair is assumed to mark the location 

of a pair of redeye defects in the resized sub-color-image. 

The result is a score P^ which indicates the likelihood Finally, the positions of the left and right redeye defects 

that a candidate redeye pixel is actually part of a redeye in the original color image are calculated based on the 

defect in the image. This score is defined by position of the left and right candidate redeye pixels in the 

50 resized sub-color-image using the relations 

Its value is in the range of 0.0 to 1 .0. The figure of merit P^^ ^ p-Pad ^ ^^^^ _ j 

is calculated for both the left and the right candidate redeye Spraaii^ 

pixels in a pair. The average of these two values is given by i-Pad 

55 ^ ~ r + RowoMoui - 1 

•^pracate 

p _ 'J" 

rpar - 2 

where p and 1 are the column and row of the left candidate 
redeye pixel in the resized sub-color-image and p' and I' are 
The pair of candidate redeye pixels for which P^^^ is the 60 the corresponding positions in the original color image S28. 
largest is referred to as the best pair of candidate redeye It sometimes happens that two different skin colored 
pixels S20f. If P^,y exceeds the threshold MinEyeScore regions after being fitted to an ellipse will overlap or be very 
equal to 0.05, then the program processes further. Otherwise, close together. This may result in the same redeye pair being 
the program concludes that a pair of redeyes is not present found twice or the detection of two redeye pairs that are too 
in the sub-color-image S20g. 65 close together for both to be truly a pair of redeyes. For this 

It is important to minimize the possibility that the best pair reason, after all the redeye pairs in the color image have been 
of candidate redeye pixels that are not part of a pair of eyes located it is determined if any two pairs have redeye 
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locations less than MinlnterpairEyeDistance equal to 20 
pixels apart. If this is the case the pair with the lower score 
is climiDatcd S30. 
We claim: 

1. A computer program product for detecting eye color 5 
defects of a subject in an image due to flash illumination, 
comprising: 

a computer readable storage medium having a Computer 
program stored thereon for performing the steps of: 

(a) searching an entire digital image for detecting one jq 
or more skin colored regions in the digital image that 
have a characteristic of a face; 

(b) resizing each skin colored region based on one or 
more predetermined facial dimensions to form one or 
more resized skin colored regions; ^5 

(c) searching the resized skin colored regions for 
groups of pixels with color characteristic of redeye 
defect; and 

(d) correcting color of the pixels based on a location of 
redeye defect found in step (c). 20 

2. The computer program product as in claim 1, wherein 
step (a) includes: 

(al) segmenting the digital image into continuous regions 
of uniform color and assigning a score indicating 
probability that the region corresponds to skin for 25 
forming a candidate skin region. 

3. The computer program product as in claim 2, wherein 
step (al) includes merging two or more candidate skin 
regions based on their similarity of color and degree of 
connectivity. 30 

4. The computer program product as in claim 2, wherein 
step (al) includes determining probability that the candidate 
region is a face based on its shape. 

5. A computer program product for detecting eye color 
defects of a subject in an image due to flash illumination, 35 
comprising: 

a computer readable storage medium having a computer 
program stored thereon for performing the steps of: 

(a) searching of an entire digital image for skin-colored 
regions having groups of pixels with color charac- 40 
teristic of redeye defect for forming candidate redeye 
defect, said skin colored regions having a character- 
istic of a face; 

(b) evaluating the groups of pixels relative to one or 
more properties of a face to determine whether a pair 45 
of the candidate redeye defects is consistent with the 
size expected of an eye based on distance between 
the pair of candidate redeye defects; and 

(c) correcting color of the pixels based on a location of 
redeye defect found in step (b). 50 

6. The computer program product as in claim 5, wherein 
the match of step (b) further includes determining for a pair 
of candidate redeye defects the tiU expected of an eye based 
on the tilt of the pair of candidate redeye defects. 

7. The computer program product as in claim 5, wherein 55 
the match of step (b) includes correlating a region around the 
candidate redeye defects with an eye template. 

8. The computer program product as in claim 7, wherein 
the match of step (b) includes correlating the region around 
the candidate redeye defects with individual zones of the eye 60 
template. 

9. A computer program product for detecting eye color 
defects of a subject in an image due to flash illumination, 
comprising: 

a computer readable storage medium having a computer 65 
program stored thereon for performing the steps of: 
(a) detecting skin colored regions in a digital image; 
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(b) resizing each skin colored region based on one or 
more predetermined facial dimensions to form one or 
more resized skin colored regions; 

(c) searching the resized skin colored regions for 
groups of pixels with color characteristic of redeye 
defect for forming a candidate redeye defect; 

(d) evaluating the groups of pixels relative to one or 
more properties of a face to determine whether a pair 
of the candidate redeye defects is consistent with a 
facial property of an eye pair; and 

(e) correcting color of the pixels based on a location of 
the redeye defect. 

10. The computer program product as in claim 9, wherein 
step (a) includes: (al) segmenting the digital image into 
continuous regions of uniform color and assigning a score 
indicating probabihty that the region corresponds to skin for 
forming a candidate skin region. 

11. The computer program product as in claim 9, wherein 
step (al) includes merging two or more candidate skin 
regions based on their similarity of color and degree of 
connectivity. 

12. The computer program product as in claim 10, 
wherein step (al) includes determining probability that the 
candidate region is a face based on its shape. 

13. The computer program product as in claim 9, wherein 
the match of step (d) further includes determining for a pair 
of candidate redeye defects the size and tUt expected of an 
eye based on distance between and the tilt of the pair of 
candidate redeye defects. 

14. The computer program product as in claim 13, 
wherein the match of step (d) includes correlating the region 
around the candidate redeye defects with individual zones of 
the eye template. 

15. The computer program product as claimed in claim 1 
wherein resizing in step (b) is based on a predetermined 
distance between the eyes of a typical face. 

16. The computer program product as claimed in claim 15 
wherein resizing in step (b) is based on a predetermined ratio 
between the width of a typical face and the distance between 
the eyes of a typical face. 

17. The computer program product as claimed in claim .1 
wherein step (c) further comprises comparing each redeye 
defect to one or more facial properties of the resized skin 
colored regions. 

18. The computer program product as claimed in claim 17 
wherein each redeye defect is compared to an eye size 
expected for the resized skin colored region. 

19. The computer program product as claimed in claim 1 
wherein step (c) further comprises detecting pairs of redeye 
defects by comparing each redeye defect in the pair to one 
or more facial properties of the resized skin colored regions. 

20. The computer program product as claimed in claim 19 
wherein the distance between each redeye defect in a pair is 
compared to a distance expected for the resized skin colored 
region. 

21. The computer program product as claimed in claim 19 
wherein each redeye defect in the pair is compared to an eye 
size expected for the resized skin colored region. 

22. A computer program product for detecting eye color 
defects of a subject in an image due to flash illumination, 
comprising: 

a computer readable storage medium having a computer 
program stored thereon for performing the steps of: 
(a) searching of an entire digital image for skin-colored 
regions having groups of pixels with color charac- 
teristic of redeye defect for forming candidate redeye 
defect, said sldn colored regions having a character- 
istic of a face; 
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(b) evaluating the groups of pixels relative to the 
determined size of a face to determine the extent to 
which the size of a candidate redeye defect matches 
an expected size of an eye pupil; and 

(c) correcting color of the pixels based on the evalua- 
tion of redeye defect performed in step (b). 

23. The computer program product as in claim 22, 
wherein the evaluation of step (b) includes correlating a 
region aroimd the candidate redeye defects with an eye 
template. 

24. A computer program product for detecting eye color 
defects of a subject in an image due to flash illumination, 
comprising: 

a computer readable storage medium having a computer 
program stored thereon for performing the steps of: 

(a) searching of an entire digital image for skin-colored 
regions having groups of pixels with color charac- 
teristic of redeye defect for forming candidate redeye 
defect, said sldn colored regions having a character- 
istic of a face; 

(b) evaluating the groups of pixels relative to the size 20 
of a face to determine the extent to which the 
distance between a pair of the candidate redeye 
defects matches an expected distance between a pair 

of eyes; and 

(c) correcting color of the pixels based on a location of 25 
redeye defect found in step (b). 

25. The computer program product as in claim 24, 
wherein the evaluation of step (b) includes correlating a 
region aroimd the candidate redeye defects with an eye 
template. 

26. A computer program product for detecting eye color 
defects of a subject in an image due to flash illumination, 
comprising: 

a computer readable storage medium having a computer 
program stored thereon for performing the steps of: 

(a) detecting groups of pixels with color characteristic 
of redeye defect for forming a pair of candidate 
redeye defects; 

(b) evaluating the groups of pixels relative to a distance 
between the redeye defects and a tilt of the redeye 
defeOs to determine the extent to which each can- 
didate redeye defect is part of an eye; and 

(c) correcting color of the pixels based on the evalua- 
tion of redeye defect found in step (b). 

27. The computer program product as in claim 26, 
wherein the evaluation of step (b) includes correlating a 
region aroimd the candidate redeye defects with an eye 
template. 

28. A computer program product for detecting eye color 
defects of a subject in an image due to flash illumination, 
comprising: 

a computer readable storage medium having a computer 
program stored thereon for performing the steps of: 

(a) searching of an entire digital image for skin-colored 
regions having groups of pixels with color charac- 55 
teristic of redeye defect for forming candidate redeye 
defect, said skin colored regions having a character- 
istic of a face; 

(b) evaluating the groups of pixels relative to one or 
more properties of a face to determine whether a pair 50 
of the candidate redeye defects is consistent with the 
tilt expected of an eye based on the tilt of the pair of 
candidate redeye defects; and 

(c) correcting color of the pixels based on a location of 
redeye defect found in step (b). 

29. The computer program product as in claim 28, 
wherein the match of step (b) further includes determining 
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for a pair of candidate redeye defects the size expected of an 
eye based on distance between the pair of candidate redeye 
defects. 

30. The computer program product as in claim 28, 
wherein the match of step (b) includes correlating a region 
aroimd the candidate redeye defects with an eye template. 

31. The computer program product as in claim 30, 
wherein the match of step (b) includes correlating the region 
around the candidate redeye defects with individual zones of 
the eye template. 

32. A method for detecting eye color defects of a subject 
in an image due to flash illumination, said method compris- 
ing the steps of: 

(a) searching an entire digital image for detecting one or 
more skin colored regions in the digital image that have 
a characteristic of a face; 

(b) resizing each skin colored region based on one or more 
predetermined facial dimensions to form one or more 
resized skin colored regions; 

(c) searching the resized skin colored regions for groups 
of pixels with color characteristic of redeye defect; and 

(d) correcting color of the pixels based on a location of 
redeye defect found in step (c). 

33. The method as in claim 32, wherein step (a) includes: 
(al) segmenting the digital image into continuous regions of 
uniform color and assigning a score indicating probability 
that the region corresponds to skin for forming a candidate 
skin region. 

34. The method as in claim 33, wherein step (al) includes 
merging two or more candidate skin regions based on their 
similarity of color and degree of connectivity. 

35. The method of claim 33, wherein step (al) includes 
determining probability that the candidate region is a face 
based on its shape. 

36. A method for detecting eye color defects of a subject 
in an image due to flash illumination, said method compris- 
ing the steps of: 

(a) searching of an entire digital image for skin-colored 
regions having groups of pixels with color character- 
istic of redeye defect for forming candidate redeye 
defect, said skin colored regions having a characteristic 
of a face; 

(b) evaluating the groups of pixels relative to one or more 
properties of a face to determine whether a pair of the 
candidate redeye defects is consistent with the size 
expected of an eye based on distance between the pair 
of candidate redeye defects; and 

(c) correcting color of pixels based on a location of redeye 
defect found in step (b). 

37. The method as in claim 36, wherein the match of step 
(b) further includes determining for a pair of candidate 
redeye defects the tilt expected of an eye based on distance 
between and the tilt of the pair of candidate redeye defects. 

38. The method as in claim 36, wherein the match of step 
(b) includes correlating a region around the candidate redeye 
defects with an eye template. 

39. The method as in claim 38, wherein the match of step 
(b) includes correlating the region around the candidate 
redeye defects with individual zones of the eye template. 

40. A method for detecting eye color defects of a subject 
in an image due to flash illumination, said method compris- 
ing the steps of: 

(a) detecting skin colored regions in a digital image; 

(b) resizing each skin colored region based on one or more 
predetermined facial dimensions to form one or more 
resized skin colored regions; 
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(c) searching the resized skin colored regions for groups 
of pixels with color characteristic of redeye defect for 
forming a candidate redeye defect; 

(d) evaluating the groups of pixels relative to one or more 
properties of a face to determine whether a pair of the 5 
candidate redeye defects is consistent with a facial 
property of an eye pair; and 

(e) correcting color of the pixels based on a location of the 
redeye defect. 

41. The method as in claim 40, wherein step (a) includes: 
(al) segmenting the digital image into continuous regions of 
uniform color and assigning a score indicating probability 
that the region corresponds to skin for forming a candidate 
skin region. 

42. The method as in claim 40, wherein step (al) includes 
merging two or more candidate skin regions based on their 
similarity of color and degree of connectivity. 

43. The method as in claim 41, wherein step (al) includes 
determining probability that the candidate region is a face 
based on its shape. 

44. The method as in claim 40, M^erein the match of step 
(d) further includes determining for a pair of candidate 
redeye defects the size and lilt expected of an eye based on 
distance between and the tilt of the pair of candidate redeye 
defects. 

45. The method as in claim 44, wherein the match of step 
(d) includes correlating the region around the candidate 
redeye defects with individual zones of the eye template. 

46. The method as claimed in claim 32 wherein resizing 
in step (b) is based on a predetermined distance between the 
eyes of a typical face. 

47. The method as claimed in claim 46 wherein resizing 
in step (b) is based on a predetermined ratio between the 
width of a typical face and the distance between the eyes of 
a typical face. 

4^. The method as claimed in claim 32 wherein step (c) 
further comprises comparing each redeye defect to one or 
more facial properties of the resized skin colored regions. 

49. The method as claimed in claim 48 wherein each 
redeye defect is compared to an eye size expected for the 
resized skin colored region. 

50. The method as claimed in claim 32 wherein step (c) 
further comprises detecting pairs of redeye defects by com- 
paring each redeye defect in the pair to one or more facial 
properties of the resized skin colored regions. 

51. The method as claimed in claim 50 wherein the 
distance between each redeye defect in a pair is compared to 
a distance expected for the resized skin colored region. 

52. The method as claimed in claim 50 wherein each 
redeye defect in the pair is compared to an eye size expected 
for the resized skin colored region. 

53. A method for detecting eye color defects of a subject 
in an image due to flash illumination, said method compris- 
ing the steps of: 

(a) searching of an entire digital image for skin-colored 
regions having groups of pixels with color character- 
istic of redeye defect for forming candidate redeye 
defect, said skin colored regions having a characteristic 
of a face; 

(b) evaluating the groups of pixels relative to the deter- 
mined size of a face to determine the extent to which 
the size of a candidate redeye defect matches an 
expected size of an eye pupil; and 

(c) correcting color of the pixels based on the evaluation 65 
of redeye defect performed in step (b). 
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54. The method as in claim 53, wherein the evaluation of 
step (b) includes correlating a region around the candidate 
redeye defects with an eye template. 

55. A method for detecting eye color defects of a subject 
in an image due to flash illumination, said method compris- 
ing the steps of: 

(a) searching of an entire digital image for skin-colored 
regions having groups of pixels with color character- 
istic of redeye defect for forming candidate redeye 
defect, said skin colored regions having a characteristic 
of a face; 

(b) evaluating the groups of pixels relative to the size of 
a face to determine the extent to which the distance 
between a pair of the candidate redeye defects matches 
an expected distance between a pair of eyes; and 

(c) correcting color of the pixels based on a location of 
redeye defect found in step (b). 

56. The method as in claim 55, wherein the evaluation of 
step (b) includes correlating a region around the candidate 
redeye defects with an eye template. 

57. A method for detecting eye color defects of a subject 
in an image due to flash illumination, said method compris- 
ing the steps of: 

(a) detecting groups of pixels with color characteristic of 
redeye defect forming a pair of candidate redeye 
defects; 

(b) evaluating the groups of pixels relative to a distance 
between the redeye defects and a tilt of the redeye 
defects to determine the extent to which each candidate 
redeye defect is part of an eye; and 

(c) correcting color of the pixels based on the evaluation 
of redeye defect foimd in step (b). 

58. The method as in claim 57, wherein the evaluation of 
step (b) includes correlating a region around the candidate 
redeye defects with an eye template. 

59. A method for detecting eye color defects of a subject 
in an image due to flash illumination, said method compris- 
ing the steps of: 

(a) searching of an entire digital image for skin-colored 
regions having groups of pixels wath color character- 
istic of redeye defect for forming candidate redeye 
defect, said skin colored regions having a characteristic 
of a face; 

(b) evaluating the groups of pixels relative to one or more 
properties of a face to determine whether a pair of the 
candidate redeye defects is consistent with the tilt 
expected of an eye based on the tilt of the pair of 
candidate redeye defects; and 

(c) correcting color of the pixels based on a location of 
redeye defect found in step (b). 

60. The method as in claim 59, wherein the match of step 
(b) further includes determining for a pair of candidate 
redeye defects the size expected of an eye based on distance 
between the pair of candidate redeye defects. 

61. The method as in claim 59, wherein the match of step 
(b) includes correlating a region around the candidate redeye 
defects with an eye template. 

62. The method as in claim 61, wherein the match of step 
(b) includes correlating the region around the candidate 
redeye defects with individual zones of the eye template. 
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